GAO 


United States Government Accountability Office _ 

Report to Congressional Committees 


F-35 SUSTAINMENT 

DOD Needs a Plan to 
Address Risks 
Related to lts Central 
Logistics System 


GAO-16-439 





April 2016 


GAD 

Highlights 

Highlights of GAO-16-439, a report to 
congressional committees 


F-35 SUSTAINMENT 

DOD Needs a Plan to Address Risks Related to lts 
Central Logistics System 


Why GAO Did This Study 

The F-35 is the most ambitious and 
expensive weapon system in DOD’s 
history, with sustainment oosts 
comprising the vast majority of DOD’s 
$1.3 trillion oost estimate. Central to 
F-35 sustainment is ALIS—a complex 
system supporting operations, mission 
planning, supply-chain management, 
maintenance, and other processes. 
The F-35 program is approaching 
several key milestones: the Air Force 
and Navy are to declare the ability to 
operate and deploy the F-35 in 2016 
and 2018 respectively, and full-rate 
production of the aircraft is to begin in 
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that GAO review the F-35’s ALIS. This 
report assesses, among other things, 
the extent to which DOD has (1) a plan 
to ensure that ALIS is fully functional 
as key program milestones approach 
and (2) credibly and accurately 
estimated ALIS oosts. GAO reviewed 
F-35 program documentation, 
interviewed officials, and conducted 
focus groups with ALIS users. 
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including that DOD develop a plan to 
address ALIS risks, and conduct 
certain analyses and include historical 
data to improve its ALIS cost estimate. 
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What GAO Found 

The Department of Defense (DOD) is aware of risks that could affect the F-35’s 
Autonomie Logistics Information System (ALIS), but does not have a plan to 
ensure that ALIS is fully functional as key program milestones approach. ALIS 
users, including pilots and maintainers, in GAO’s focus groups identified benefits 
of the system, such as the incorporation of multiple functions into a single 
system. However, users also identified several issues that could result in 
operational and Schedule risks. These include the following: 

• ALIS may not be deployable: ALIS requires server connectivity and the 
necessary infrastructure to provide power to the system. The Marine Corps, 
which often deploys to austere locations, declared in July 2015 its ability to 
operate and deploy the F-35 without conducting deployability tests of ALIS. A 
newer version of ALIS was put into operation in the summer of 2015, but 
DOD has not yet completed comprehensive deployability tests. 

• ALIS does not have redundant infrastructure: ALIS’s current design 
results in all F-35 data produced across the U.S. fleet to be routed to a 
Central Point of Entry and then to ALIS’s main operating unit with no backup 
system or redundancy. If either of these fail, it could take the entire F-35 fleet 
offline. 

DOD is taking some steps to address these and other risks such as resolving 
smaller ALIS functionality issues between major software upgrades and 
considering the procurement of additional ALIS infrastructure but the department 
is attending to issues on a case-by-case basis. DOD does not have a plan that 
prioritizes ALIS risks to ensure that the most important are expediently 
addressed and that DOD has a fully functional ALIS as program milestones draw 
close. By continuing to respond to issues on a case-by-case basis rather than in 
a holistic manner, there is no guarantee that DOD will address the highest risks 
by the start of full-rate production in 2019, and as a result, DOD may encounter 
further Schedule and development delays, which could affect operations and 
potentially lead to cost increases. 

DOD has estimated total ALIS oosts to be about $16.7 billion over the F-35’s 
56-year life cycle, but performing additional analyses and including historical cost 
data would increase the credibility and accuracy of DOD’s estimate. GAO’s cost 
estimating best practices state that cost estimates should include uncertainty 
analyses to determine the level of uncertainty associated with the estimate in 
order to be credible. In addition, credible cost estimates should include sensitivity 
analyses to examine how changes to individual assumptions and inputs affect 
the estimate as a whole. DOD's guidance does not require the department to 
perform these analyses for ALIS, and DOD officials stated that they have not 
done so in part because ALIS constitutes less than 2 percent of the F-35’s 
estimated total sustainment oosts. Program officials said that if ALIS is not fully 
functional, the F-35 could not be operated as frequently as intended, but a DOD- 
commissioned plan found that Schedule slippage and functionality problems with 
ALIS could lead to $20-100 billion in additional oosts. Without uncertainty and 
sensitivity analyses, it is unclear how ALIS can affect oosts. GAO also found that 
using historical cost data would make DOD’s cost estimate more accurate. 
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441 G St. N.W. 
Washington, DC 20548 


U.S. GOVERNMENT ACCOUNTABILITY OFFICE 


April 14, 2016 
Congressional Committees 

The Department of Defense (DOD) calls the F-35 Lightning II the 
backbone of U.S. air combat superiority for decades to come, as it plans 
for the aircraft to replace the legacy tactical fighter fleets of the Air Force, 
Navy, and Marine Corps. With estimated total ownership costs of just 
under $1.3 trillion over its 56-year life cycle, the F-35 is the most 
ambitious and costly program in DOD’s history. 1 The oost to sustain the 
aircraft—estimated to be just under $900 billion—is the primary cost 
driver for the F-35 program. Central to F-35 sustainment is the Autonomie 
Logistics Information System (ALIS)—a complex system that supports 
operations, mission planning, supply-chain management, maintenance, 
and other processes. A DOD official has described ALIS as the brains of 
the aircraft, calling it one of the three major components that make up the 
F-35, along with the airframe and engine. However, ALIS has 
experienced developmental issues, including system functionality 
problems, and Schedule delays that have put key performance 
parameters, such as aircraft availability and flying missions, at risk. The 
F-35 program is approaching several key milestones: the Air Force and 
Navy are to declare the ability to operate and deploy the aircraft (“initial 
operational capability”) in 2016 and 2018 respectively, and full-rate 
production decision of the program is planned for 2019. 2 Recognizing that 
a fully functional ALIS is critical to the program’s overall success, in 
October 2015, the F-35 executive program officer testified before 
Congress that ALIS is one of the most significant technical and Schedule 
risks to the program. 3 


^he $1.3 trillion dollar total ownership cost and all other cost estimates throughout this 
report are based on the 2014 F-35 program office cost estimate. 

2 The Marine Corps declared initial operational capability in July 2015. The full-rate 
production is a decision following the completion of operational testing, to scale up 
production and fielding. During full-rate production, the remaining production or 
deployment of the product is completed, leading to full operational capability or full 
deployment. 

3 F-35 Lightning II Program Update, testimony before the Subcommittee on Tactical Air 
and Land Forces, House Committee on Armed Services H. Comm. on Armed Services, 
Subcomm. on Tactical Air and Land Forces, 114 th Cong., 1 st session (October21,2015). 
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We have reported on DOD’s acquisition of the F-35 for many years (see 
the Related GAO Products section at the end of this report) and more 
recently, on F-35 sustainment issues. In September 2014, we found that: 
(1) DOD had not fully addressed several issues—including ALIS-related 
problems—that posed risks to the long-term affordability and operational 
readiness of the program; (2) DOD had not linked established 
sustainment affordability targets with military service budgets, meaning 
that the targets may not be representative of what the services could 
actually afford; and, (3) DOD had weaknesses with respect to a few cost 
assumptions and did not include all necessary analyses to make its 
sustainment cost estimates fully reliable. 4 With respect to ALIS, we 
reported that although DOD had originally planned to have a fully 
functional and effective logistics system by 2017—7 years after originally 
intended—to ensure operational readiness and availability, DOD did not 
have a performance measurement process, with metrics and targets, to 
determine and address performance issues with ALIS. We recommended 
that DOD establish a performance measurement process for ALIS that 
included, but was not limited to, performance metrics and targets that are 
based on intended behavior of the system in actual operations and that 
tie system performance to user requirements. DOD concurred with our 
recommendation and, as of January 2016, was in the process of 
developing a performance-measurement process for ALIS. 

Citing the importance of a fully functional ALIS to the overall performance 
on the F-35, the Subcommittee on Readiness, House Armed Services 
Committee, asked us to review the status, costs, and risks associated 
with ALIS. Subsequently, the National Defense Authorization Act for 
Fiscal Year 2016 included a provision that we report on ALIS, including 
an assessment of the capability of the program to address performance 
problems within planned resources. 5 This report assesses the extent to 
which DOD has (1) a plan to ensure that ALIS is fully functional as key 
program milestones approach, (2) credibly and accurately estimated 
costs associated with ALIS, and (3) developed a plan to manage ALIS 
training for users. 


4 GAO, F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and 
Improved Cost Estimates, GAO-14-778 (Washington, D.C.: Sept. 23, 2014). 

5 Pub. L. No. 114-92, § 242 (2015). 
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To determine the extent to which DOD has a plan to ensure ALIS is fully 
functional by key program milestones—which include the Air Force and 
Navy initial operational capability declarations and the start of full-rate 
production forthe F-35 program—we reviewed the F-35 program’s 
sustainment documents and conducted site visits at a nongeneralizable 
sample of five F-35 operational and training sites: Eglin Air Force Base, 
Marine Corps Air Base Yuma, Luke Air Force Base, Edwards Air Force 
Base, and Nellis Air Force Base. We selected these sites in consultation 
with service officials to ensure we obtained perspectives from primary 
operational and testing sites. 6 For the purposes of this review, we focused 
solely on the U.S. F-35 fleet. During these visits, we also held 17 
nongeneralizable focus group sessions with a range of ALIS users (e.g., 
maintainers, pilots) from all three services to obtain information on the 
operability and deployability of ALIS, and how any ALIS issues may pose 
risks to F-35 operations and sustainment. These focus groups also 
included contractor personnel responsible for administering ALIS and 
providing training at these sites. Approximately a total of 120 people 
participated in these focus groups. Additionally, we interviewed key DOD 
and contractor officials. We evaluated the information we obtained for 
consistency with GAO’s Schedule Assessment Guide and DOD’s System 
Engineering Guide for System of Systems that provide guidance and best 
practices on how, prior to meeting key milestones, a plan should be 
developed to address specific risks that may be associated with major 
weapon acquisitions. 7 

To determine the extent to which DOD has credibly and accurately 
estimated ALIS costs, we evaluated the reliability of DOD’s estimate of 
ALIS costs contained in the most recent F-35 program office estimate of 


6 These locations included Air Force and Marine Corps bases. In addition, although we did 
not visit a Navy installation, we spoke with Navy personnel working in the F-35 program at 
one of these locations. 

7 GAO, Schedule Assessment Guide, GAO-12-120G (Washington, D.C.: May 2012) and 
Department of Defense Systems Engineering Guide for System of Systems (Washington, 
D.C.: August 2008). 
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F-35 operating and support (O&S) oosts 8 using characteristics contained 
in GAO’s Cost Estimating and Assessment Guide. 9 Forthe purposes of 
this review, we conducted a limited assessment and used two of the four 
general characteristics of sound cost estimating included in this guide: 
being credible and accurate. We chose to use these two characteristics 
because ALIS is not a stand-alone weapon system program with its own 
cost estimate, and its projected costs are included in the larger F-35 cost 
estimate. Therefore, we only assessed ALIS costs and not an entire life- 
cycle cost estimate, and determined that it would not be appropriate to 
assess whether the estimate was comprehensive or well-documented 
because ALIS costs represent one element of the total F-35 cost 
estimate—less than 2 percent of projected F-35 sustainment costs. To 
determine whether the credible and accurate characteristics were met, we 
reviewed cost-estimating documentation, including data sources, 
assumptions, and cost models, and we interviewed cost-estimating 
officials from the F-35 program office and the Office of the Director for 
Cost Assessment and Program Evaluation (CAPE). We also interviewed 
other officials from the F-35 program office and service headquarters to 
discuss the cost effects of ALIS Schedule delays and development issues. 
We found the data to not be fully reliable, which we discuss in further 
detail in the report and in appendix II. 

Finally, to determine the extent to which DOD has developed a plan to 
manage ALIS training for users, we reviewed key documentation related 
to ALIS and F-35 training, and used information from our focus-group 
sessions with a range of ALIS users from all three services to obtain 
information on the current state of ALIS training. We also interviewed key 
DOD and contractor officials. We evaluated all of the information we 
received using GAO-developed and industry best practices for 


8 The F-35 program office updates its cost estimate annually. The most recent estimate 
that was completed, approved, and promulgated during our review was the 2014 estimate, 
released in spring 2015. The 2015 estimate is currently being reviewed within the program 
office and is expected to be finalized after issuance of this report in spring 2016. The 
Office of the Director for Cost Assessment and Program Evaluation (CAPE) conducted a 
separate estimate in 2013 but has not updated it since then. In September 2014, we 
reported on the reliability of CAPE’s estimate. See GAO-14-778. 

9 GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and 
Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). 
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information-technology training, and guidance on assessing strategie 
training and development efforts in the federal government. * 1 2 3 * * * * * * 10 

We conducted this performance audit from April 2015 to April 2016 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficiënt, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. Appendix I provides a more 
detailed description of our scope and methodology. 


Background 


F-35 Program and Costs The F-35 Lightning II program, also known as the Joint Strike Fighter, is a 

joint, multinational acquisition intended to develop and field a family of 
next-generation strike fighter aircraft for the United States Air Force, 

Navy, and Marine Corps, and eight international partners. 11 According to 
DOD, there will be three variants of the F-35: 

1. The conventional takeoff and landing (CTOL) variant, designated the 
F-35A, will be a multirole, stealthy strike aircraft replacement for the 
Air Force’s F-16 Falcon and the A-10 Thunderbolt II aircraft, and will 
complement the F-22A Raptor (see fig. 1). 

2. The short takeoff and vertical landing (STOVL) variant, designated the 
F-35B, will be a multirole, stealthy strike fighter that will replace the 
Marine Corps’ F/A-18C/D Hornet and AV-8B Harrier aircraft. 

3. The carrier-suitable variant (CV), designated the F-35C, will provide 
the Navy a multirole, stealthy strike aircraft to complement the F/A-18 


10 GAO, Information Technology Training: Practices of Leading Private-Sector Companies, 

GAO-03-390 (Washington, D.C.: Jan. 31,2003) and Human Capital: A Guide for 

Assessing Strategie Training and Development Efforts in the Federal Government, 

GAO-04-546G (Washington, D.C.: March 2004). 

11 The international partners are the United Kingdom, Italy, the Netherlands, Turkey, 

Canada, Australia, Denmark, and Norway. These nations contributed funds for system 

development and all but Canada and Denmark have signed agreements thus far to 
procure aircraft. In addition, Israël, Japan, and South Korea have signed on as foreign 

military sales customers. 
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E/F Super Hornet. The Marine Corps will also field a limited number of 
F-35C CVs. 

Lockheed Martin is the primary aircraft contractor and Pratt & Whitney is 
the engine contractor. 


Figure 1: F-35A Air Force Conventional Takeoff and Landing Variant (CTOL) 



Source: Department of Defense. | GAO-16-439 


Although the acquisition costs of the F-35 program are about $400 billion, 
the most significant cost driver for the program is sustainment. The F-35 
O&S costs are those incurred from the initial system deployment through 
the end of system operations, and include all costs of operating, 
maintaining, and supporting the fielded system. The F-35 program office 
develops an annual estimate for the O&S costs of maintaining and 
supporting the F-35 for 56 years. 12 In its most recent estimate (2014), the 
program office estimates that it will cost about $891 billion to sustain the 
entire F-35 fleet over its life cycle. 13 


Autonomie Logistics 
Information System 
(ALIS): Primary Logistics 
Tool for the F-35 


The Autonomie Logistics Information System (ALIS) is a system of 
systems that serves as the primary logistics tooi to support F-35 
operations, mission planning, and sustainment. ALIS helps maintainers 
manage tasks including aircraft health and diagnostics, supply-chain 
management, and necessary maintenance events. Lockheed Martin is 
the prime contractor for ALIS and is responsible for developing and 


12 Each aircraft has a 30-year service life, but the 2014 estimate spans 56 years. 
13 The $891 billion dollar estimate is in then-year dollars, adjusted for inflation. 
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managing the capabilities of the system, as well as developing training 
materials for ALIS users. 14 

According to DOD, ALIS will be co-located with F-35 aircraft both at U.S. 
military installations and in theater to support missions and assist with 
maintenance and resource allocation. ALIS consists of the overarching 
system, the applications housed within it, and the some of the network 
infrastructure required to provide global integrated and autonomie support 
of the F-35 fleet. It comprises both hardware and software. The hardware 
consists of three main components: 

• The Autonomie Logistics Operating Unit (ALOU): The ALOU is the 
computer server that all F-35 data ultimately are sent through and it 
supports Communications with and between the government and the 
contractor’s systems. 

• The Central Point of Entry (CPE): The CPE is configured to provide 
software and data distribution for the entire F-35 fleet in the United 
States, enables interoperability with national (government) systems at 
the country level, and enables ALIS data connectivity between bases. 
Each international partner operating F-35 aircraft is expected to have 
its own CPE at other locations. 

• The Standard Operating Unit (SOU): SOUs provide all ALIS 
capabilities to support flying, maintenance, and training. They also 
provide access to applications to operate and sustain the aircraft. 

As of February 2016, there was one operational ALOU and CPE within 
the United States. Each F-35 operating and testing site in the United 
States has a varying number of SOUs depending on the site’s number of 
aircraft and squadrons, and there are two versions: SOU VI and SOU V2. 
The main difference between the two SOUs is that SOU V2 was designed 
to better meet participants’ deployability requirements. While SOU VI 
was housed in two 1,600-pound server racks, SOU V2 was designed to 
have its components fit into transit cases that are two-man portable, each 
weighing approximately 200 pounds. DOD is planning to have at least 
one SOU accompany each F-35 squadron. The services organize their 
squadrons differently but squadron sizes generally range from 10 to 24 
aircraft. The F-35 Operational Requirements Document, which originated 
in March 2000 and contains the performance and operational parameters 
for the concept of the F-35, calls for an incremental development of F-35 


14 

For the purposes of this report, Lockheed Martin will be referred to as the “contractor.” 
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capabilities by aircraft software blocks and phased software releases 
during the system development and demonstration phase. 15 This is 
concurrent to the production and fielding of small volumes of aircraft 
during low-rate initial production. 16 ALIS’s software development is 
anticipated to be completed after two more of its versions are released, 
the first of which will support the Air Force declaring initial operational 
capability in summer 2016. The current fielding status of ALIS is 
illustrated in figure 2, which also includes the dates for when the next 
software upgrade—ALIS Version 2.0.2—is to be introduced at each F-35 
site. ALIS 3.0, which is the version expected to meet the requirements 
defined in the Operational Requirements Document, is expected to have 
completed its development and commence testing by October 2017 in 
line with the end of system development and demonstration. 17 The 
release is to be fielded under the low-rate initial production contract in 
early 2018 and is intended to be fully functional by 2019. 


15 System development and demonstration, is now known as the engineering & 
manufacturing development (EMD) phase. The purpose of the EMD phase is to develop, 
build and test a product to verify that all operational and derived requirements have been 
met, and to support production or deployment decisions. Because this program acquisition 
plan was developed prior to this change, for the purposes of this report we will continue to 
use the term “system development and demonstration.” 

16 l_ow-rate initial production establishes the initial production base for the system or 
capability increment, provides an efficiënt ramp up to full-rate production, and maintains 
continuity in production pending operational test and evaluation completion. For the F-35, 
low-rate initial production is taking place during the system development and 
demonstration phase. 

17 Department of Defense, Joint Strike Fighter, Operational Requirements Document 
(Washington, D.C.: Aug.19, 2008). 
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Figure 2: Location of United States ALIS Hardware and Planned ALIS 2.0.2 Software Upgrade Dates 


Ogden Depot 

Utah 

• 1 SOU VI 

• Proposed software upgrade 
August 2016 

Nellis Air Force Base 

Nevada 

• 2 SOU VI 

• 1 SOU V2 

• Proposed software upgrade 
October 2016 

Edwards Air Force Base 

California 

• 4 SOU VI 

• Proposed software upgrade 
October 2016 


Marine Corps Air 
Station Yuma 

Arizona 



Hill Air Force Base 

Utah 

• 1 SOU VI 

• Proposed software 
upgrade: July 2016 



• 1 SOU VI 

• 2 SOU V2 

• Proposed software upgrade: 
September 2016 


Luke Air Force Base 

Arizona 

•5 SOU VI 
• Proposed software 
upgrade: 

November 2016 



® 

Fort Worth 

Texas 

SOU V2 
oposed 
software 
upgrade: 
July 2016 


se 

Florida 

• 3 SOU VI 

• Proposed 
software 
upgrade: 
August 2016 


USS Wasp 

Norfolk, Virginia 

• 1 SOU VI 

• Proposed 
software 
upgrade: 
November 
2016 


Cherry Point Depot 

North Carolina 

• 1 SOU VI 

• Proposed software 
upgrade: 
September 2016 

Marine Corps Air 
Station Beaufort 

South Carolina 

• 2 SOU VI 

• Proposed software 
upgrade: 

August 2016 


Source: GAO presentation of Department of Defense information. | GAO-16-439 


Notes: As of January 2016, all locations had been upgraded to ALIS 2.0.1. 

SOU Version 1 is hardware located at the squadron level and is housed in two 1,600-pound server 
racks. 


SOU Version 2 is hardware located at the squadron level and is designed to meet participants’ 
deployability requirements, The components fit into transit cases that are two-man portable and 
weight approximately 200 pounds each. 


DOD intends for ALIS software capabilities to include: operational 
planning, maintenance, supply-chain management, customer-support 
services, training, tech data, system security, and external interfaces. As 
stated earlier, ALIS is a system of systems—it comprises multiple 
software applications that perform specific functions for F-35 maintainers, 
pilots, supply personnel, and data analysts. These separate applications 
must be integrated within the system, as well as have interconnectivity 
with preexisting “legacy” information systems that are used by the 
services for other weapons system platforms. ALIS applications are being 
developed by the contractor incrementally, with some applications 
currently more functional than others. The F-35 program’s original 
requirements state that it must include a fully functional and effective 
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logistics system to ensure operational readiness and availability. As of 
December, average aircraft availability for the year—the percentage of F- 
35 aircraft capable of performing missions at a given time—was around 
50 percent, whereas DOD had expected it to be 60 percent. Figure 3 
includes major applications within ALIS, their intended purpose, and the 
F-35 program office’s assessment of the functionality status of each 
application. As figure 3 shows, according to DOD, most of the 
applications within ALIS currently have functionality issues. 
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Figure 3: Primary ALIS Applications and the F-35 Program Office’s Assessment of Their Functionality Status as of January 
2016 



Autonomie Logistics Information Systems (ALIS) is the 

primary sustainment tooi for the F-35 and is intended to 
predict maintenance and supply issues, automate logistics 
support processes, and provide decision aids to help reduce 
life-cycle sustainment oosts and improve force readiness. 



Computerized Maintenance 
Management System (CMMS) is used 
to initiate and track maintenance 
actions, Schedule of work, aircraft and 
support equipment status, access the 
JTD, and provide traceability to “as 
built” and “as maintained” configuration 
data. 


Minor issues as reported by F-35 program office 


Maintenance Vehicle Interface (MVI) 

provides an interface to the aircraft for 
the portable maintenance aid. This 
allows the maintainer to operate 
aircraft subsystems, and to retrieve 
and display aircraft status including fuel 
levels, pump status, and weapon 
status. 


Minor issues as reported by F-35 program office 


Propulsion processes the data 
downloaded from the aircraft, including 
propulsion Health Reporting Codes as 
a basis for AVD/H to generate work 
orders that are passed to CMMS. 


Minor issues as reported by F-35 program office 


Source: GAO presentation of Department of Defense analysis. | GAO-16-439 




Note: According to DOD officials, the functionality status (major, minor, no issues) of applications 
within ALIS is based on user feedback. 
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ALIS was originally scheduled to be completed fortesting in 2010, as 
shown in figure 4. However, that same year the program triggered a 
Nunn-McCurdy breach, 18 when unit cost growth exceeded critical 
thresholds. As a result, the F-35 program (including ALIS) was 
rebaselined in 2012, which established a new acquisition baseline, and 
Milestone B was recertified afterthe breach. 19 


Figure 4: ALIS Schedule History, Original Timeline, and Rebaseline 


Original scheduled completion of 
Autonomie Logistics Information 
Systems (ALIS) 3.0 for testing 


2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 




2001: Milestone B is approved and 
Lockheed Martin is selected as the 
prime contractor. 3 
The program begins development. 


T 1 

2010: F-35 2012: F-35 


program 
triggers a 
Nunn-McCurdy 
Breach b 


program is 
rebaselined, 
including ALIS 


After rebaselining, ALIS 
3.0 is scheduled to be 
completed for testing 


2015 | 2016 | 2017 [ 2018 | 2019 \ 2020-2040 


I 1 TT 

2019: Planned 
milestone decision to 
enter full-rate production 

2018: Navy is scheduled 
to declare initial 
operational capability 

2016: Air Force is scheduled to declare 
initial operational capability in August 

2015: Marine Corps declares 
initial operational capability in July 


Source: GAO analysis of Department of Defense information. | GA0-16-439 

a As part of DOD’s acquisition process, Milestone B initiates the engineering & manufacturing 
development phase. 

b Section 2433 of title 10 of the United States Code, commonly referred to as Nunn-McCurdy, requires 
DOD to notify Congress whenever a major defense acquisition program’s unit cost experiences cost 
growth that exceeds certain thresholds. This is commonly referred to as a Nunn-McCurdy breach. 
Significant breaches occur when the program acquisition unit cost or procurement unit cost increases 


18 

Section 2433 of title 10 of the United States Code, commonly referred to as Nunn- 
McCurdy, requires DOD to notify Congress whenever a major defense acquisition 
program’s unit cost experiences cost growth that exceeds certain thresholds. This is 
commonly referred to as a Nunn-McCurdy breach. Significant breaches occur when the 
program acquisition unit cost or procurement unit cost increases by at least 15 percent 
over the current baseline estimate or at least 30 percent over the original estimate. For 
breaches of the critical threshold, as was the case with the F-35 in 2010, when these unit 
costs increase at least 25 percent over the current baseline estimate or at least 50 percent 
over the original, DOD is required to take additional steps, including conducting an in- 
depth review of the program. Programs with critical breaches must be terminated unless 
the Secretary of Defense certifies to certain facts related to the program and takes other 
actions, including restructuring the program. 10 U.S.C. § 2433a. 

19 As part of DOD’s acquisition process, Milestone B provides authorization for the 
program to enter the engineering & manufacturing development phase. 
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by at least 15 percent over the current baseline estimate or at least 30 percent over the original 
estimate. For critical breaches, as was the case with the F-35 in 2010, when these unit costs increase 
at least 25 percent over the current baseline estimate or at least 50 percent over the original, DOD is 
required to take additional steps, including conducting an in-depth review of the program. Programs 
with critical breaches must be terminated unless the Secretary of Defense certifies to certain facts 
related to the program and takes other actions, including restructuring the program. 10 U.S.C. § 
2433a. 

The F-35 program is currently approaching the end of the system 
development and demonstration phase, during which ALIS has been 
developed, built, and tested to verify that operational requirements are 
being met. Concurrently, the program is in low-rate initial production. 20 In 
addition, DOD and the services are in the process of declaring major 
milestones. Specifically, the Marine Corps declared initial operational 
capability in July 2015. The Air Force is scheduled to declare initial 
operational capability in August 2016, and the Navy is scheduled to do so 
in 2018. By 2018, after all services have declared their initial operational 
capability, the program, to include ALIS, is expected to reach full 
warfighting capability. The F-35 program plans to begin full-rate 
production in 2019, and according to DOD officials, any additional 
modifications or upgrades with new capabilities to ALIS will be part of the 
program’s follow-on modernization. 21 DOD generally requires programs to 
have established sustainment and support systems, like ALIS, for the F- 
35 by full-rate production. According to the F-35 Operational 
Requirements Document, by full-rate production, all variants must be able 
to deploy rapidly, sustain high mission reliability, and sustain a high 
sortie-generation rate. 22 


90 

We have previously reported on the overlap between technology development and 
product development and production in this program’s acquisition strategy. In our prior 
work, we have identified the lack of knowledge and high levels of concurrency as major 
drivers in the significant oost and Schedule growth as well as performance shortfalls that 
the program has experienced since 2001. 

21 Follow-on modernization involves continuous incremental upgrades to the aircraft, 
including ALIS, to keep it affordable, interoperable, and operationally suitable throughout 
the life of the program. These upgrades will include common requirements, corrections to 
operational deficiencies identified in operational or developmental tests, and capabilities 
from the Operational Requirements Document. 

22 Sorties, orflights, are generated in support of testing and operations. 
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DOD Is Aware of 
Risks to ALIS 
Functionality That 
Users Have 
Identified, but Does 
Not Have a Plan to 
Prioritize and Address 
Them as Key 
Program Milestones 
Approach 


ALIS users (pilots, maintainers, administrators, and trainers) in the focus 
groups we held at five F-35 operational or testing sites identified some 
benefits of the system. For example, maintainers and administrators at 
three sites stated that they have seen ALIS’s capabilities improve over 
time as the system’s software has been upgraded. In addition, pilots and 
maintainers at three sites expressed confidence in ALIS’s future 
capabilities as the system continues to improve. Maintainers and trainers 
at three sites also found ALIS’s design—which incorporates multiple 
functions within one single system—helpful for efficiently executing tasks, 
when previously they had to work in multiple, separate systems. For 
example, trainers at Eglin Air Force Base stated that, with legacy aircraft, 
there are separate systems for tasks such as recording maintenance 
work and ordering parts. With the F-35, ALIS houses applications for 
these tasks within its system, which can be more convenient to users. In 
addition, maintainers at two sites stated that, following a recent software 
upgrade, the system processes information faster, which improves 
maintenance data input capabilities. Maintainers at three sites also told us 
that ALIS performs some tasks better than legacy systems. Specifically, 
maintainers at Eglin Air Force Base and Nellis Air Force Base explained 
that because ALIS stores information electronically, it eliminates the need 
for paper-based manuals commonly used on legacy aircraft. For instance, 
maintainers at Eglin Air Force Base noted that the technical data that they 
use to assist in aircraft repair is now stored electronically within ALIS and 
can be updated as necessary, whereas before this information was 
contained in multiple paper-based manuals, which were difficult to 
efficiently access and keep up-to-date. Most users we spoke with 


ALIS Users Identified 
Benefits but Also Key 
Risks to the System 


DOD is aware of risks that could affect ALIS but does not have a plan to 
prioritize and address them in a holistic manner to ensure that ALIS is 
fully functional as the F-35 program approaches key milestones— 
including Air Force and Navy initial operational capability declarations in 
2016 and 2018, respectively, and the start of the program’s full-rate 
production in 2019. During our focus groups, ALIS users identified some 
beneficial aspects of the system. However, they also identified a variety of 
concerns, which may result in operational and Schedule risks. The 
F-35 program office is aware of these issues, but is currently addressing 
them on a case-by-case basis. 
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recognized that ALIS is a system in development and stated that its 
immaturity was to be expected at this stage in the program. 

However, during our focus-group sessions, ALIS users also identified 
several issues, which, if not addressed, could result in operational and 
Schedule risks. Table 1 summarizes the risks reported by the majority of 
participants in our 17 focus groups. DOD is aware of these risks and, as 
discussed later, is addressing risks on a case-by-case basis. 

Table 1: ALIS Risks Identified by Users in GAO’s Focus Groups a 

Functionality issue 

Description of Autonomie Logistics Information System (ALIS) user concerns 

Deployability of ALIS 

Users are concerned about ALIS’s ability to deploy in operational environments because of the 
large Standard Operating Unit (SOU) server size and connectivity requirements. The Marine 
Corps, which often deploys to austere locations, declared initial operational capability in July 

2015 without conducting deployability tests of ALIS. Although a more deployable version of ALIS 
was put into operation in the summer of 2015, the Department of Defense (DOD) has not yet 
completed comprehensive deployability tests. 

No redundancy in its infrastructure 

Users are concerned that ALIS’s current design results in all F-35 data produced across the fleet 
to be routed to the Central Point of Entry and then to the Autonomie Logistics Operating Unit, 
with no backup system or redundancy. If either of these fail, it could take the entire F-35 fleet 
offline. 

Ability to communicate with legacy 
systems 

Users are concerned that ALIS does not have much interoperability with legacy systems. The 
services will continue using legacy systems for other weapon systems, and some of these data 
must be shared with ALIS. Because ALIS currently has little interoperability with legacy systems, 
some of this information is currently being tracked manually outside of ALIS, which is inefficiënt 
and could potentially result in data not being populated back into the system. 

Action Request 0 (AR) process is 
inefficiënt and problematic 

Users report that the current AR process does not provide transparency of all ARs submitted 
across F-35 sites, and places responsibility for resolving the requests primarily on the contractor. 

Data accuracy and accessibility 
issues 

Users report concerns about data that reside within ALIS, including errors related to missing or 
inaccurate information about parts and not being able to extract raw data to generate needed 
reports. 

Off-board Mission Support (OMS) 
and Training Management System 
(TMS) are immature 

Users report that both ALIS applications are important to their day-to-day operations, but are 
immature and do not function as intended. OMS is a key application used by pilots to conduct 
mission planning and debriefing. TMS is used by pilots and maintainers to track training 
qualifications. 

Security concerns 

Users report concerns related to transferring information between unclassified and classified 
computer servers and related to cyber security. A 2012 DOD Inspector General’s report 0 on ALIS 
also highlighted some security issues, including security accreditation and testing of hardware. 

Source: GAO | GAO-16-439 

Notes: 

a We visited five operational and training sites and convened focus groups with a range of ALIS users. 
Specifically, we convened focus group sessions with pilots, maintainers, administrators, and trainers. 
Some of the trainers and administrators were contractors. These risks were identified by ALIS users 
across the range of groupings at all five sites. The views are not generalizable to all users. 

b An AR is a concern or question raised by a customer or user about any area of the F-35 system, 
including ALIS. 
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c Department of Defense Inspector General, Audit of F-35 Lightning II Autonomie Logistics Information 
System (ALIS) (Washington, D.C.: Dec. 10, 2012). 


The risks identified by ALIS users during our focus group sessions are 

discussed in more detail below. 

• ALIS may not be able to deploy: Pilots, maintainers, and 

administrators at three of the five sites we visited are concerned about 
ALIS’s ability to deploy and function in forward locations. For example, 
users are concerned about the large server size and connectivity 
requirements, and whether the system’s infrastructure can maintain 
power and withstand a high-temperature environment. The Marine 
Corps, which often deploys to austere locations, did not conduct 
deployability tests prior to declaring initial operational capability in July 
2015. 23 ALIS’s original requirements 24 did not include specific 
deployability requirements, so the system’s original hardware design 
consisted of large, heavy racks of servers. DOD officials stated that 
the Marine Corps subsequently added specific requirements for a 
deployable system to meet its expeditionary mission needs. Although 
the more deployable version of ALIS was fielded in summer of 2015, 
DOD has yet to complete comprehensive deployability testing. In 
December 2015, the Marine Corps participated in an exercise at the 
Strategie Expeditionary Landing Field near the Marine Corps Air 
Ground Combat Center in California (also known as Twentynine 
Palms) that included a short-range, domestic deployability test of the 
system. According to DOD officials, the results were positive in that 
the Marine Corps transported the system to Twentynine Palms from 
its Yuma base and set it up within 2 hours; however, this test did not 
include long-range, overseas, ship-based, or combat scenarios. Air 
Force and Navy officials stated that they plan to conduct deployabilty 
tests prior to declaring initial operational capability over the next 2 
years; however, these officials expressed concerns over the ability of 


po 

The Marine Corps conducted F-35 tests onboard the USS Wasp prior to declaring 
operational capability in July 2015, including day and night carrier missions and 
maintenance exercises; however, these tests did not include deployability tests of ALIS. 
According to the Director of Test and Evaluation, these tests were not operationally 
representative because of the heavy use of contractor support, and lack of other types of 
aircraft sharing the flight deck. He also noted that this test used the original, 
nondeployable ALIS server. 

24 Department of Defense, Joint Strike Fighter ; Operational Requirements Document. 
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ALIS to function in austere environments and in split-squadron 
situations 25 that would require multiple deployable ALIS servers. 

• ALIS does not have redundancy in its infrastructure: ALIS users 
at three of the five sites we visited are concerned that a failure in the 
system’s current infrastructure could degrade the system and ground 
the fleet. Currently, ALIS Information, including data from all U.S. 

F-35 sites, flows from the Standard Operating Units (SOU) to a single 
national Central Point of Entry, and then to the lone Autonomie 
Logistics Operating Unit (ALOU). 26 This data flow process has no 
back up system for continuity of operations if either of these servers 
were to fail. Specifically, squadron leadership at two sites expressed 
concern about how the loss of electricity due to weather or other 
damaging situations could adversely affect fleet operations if either 
the Central Point of Entry or the ALOU went offline. DOD officials told 
us that they recognize this issue and, for short-term losses of 
connection, ALIS users are able to work offline. Program officials also 
said that they are in the early stages of trying to procure up to two 
additional ALOUs and possibly relocating the Central Point of Entry to 
another F-35 site. However, as of January 2016, DOD officials had 
just begun to explore this option and have not allocated any resources 
to support the idea. 

• ALIS does not effectively communicate with legacy aircraft 
systems: Maintainers and pilots at three of the five sites we visited 
were concerned that ALIS does not have much interoperability with 
legacy aircraft systems. For example, while ALIS was designed to 
house multiple applications within it, the Air Force, Navy, and Marine 
Corps will continue to use legacy systems to operate and maintain 
other weapon systems. The ability to share Information between ALIS 
and these legacy systems is vital due to the way the services operate. 
In particular, Marine Corps officials noted that because the service 
operates with squadrons that use data from the Navy’s legacy 


25 According to DOD officials, a split-squadron is when parts of a squadron deploy 
separately, leaving the full squadron in multiple locations. This, according to the same 
officials, is a common practice as most full squadrons do not often deploy together. 

26 There is one Central Point of Entry for all U.S. F-35 operational squadrons and aircraft, 
and another for all training squadrons and aircraft. Officials told us that the training Central 
Point of Entry does not provide redundancy. 
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system 27 and ALIS, it would be beneficial for those two systems to 
communicate with one another. DOD officials stated that the services 
are responsible for developing the software interface that can take 
information from ALIS and translate it so that it can be communicated 
to the legacy systems. However, due to the lack of interoperability 
between ALIS and legacy systems, users are being forced to track 
data outside of ALIS, which, according to maintainers, is inefficiënt 
and could potentially result in data not being populated back into the 
system. 

• Action Request process is inefficiënt and problematic: 

Maintainers at four of the five sites we visited told us that the current 
Action Request 28 (AR) process does not allow for the effective 
reporting and resolution of F-35 aircraft and ALIS issues. Personnel 
use an application within ALIS to submit an AR about any F-35 
problems, including those with ALIS itself, to the contractor for triaging 
and ultimate resolution. 29 However, these maintainers explained that 
the process is too heavily controlled by the contractor and that users 
lack visibility of ARs submitted from other F-35 sites or squadrons. 
Consequently, ALIS users have to wait for the contractor to conclude 
that multiple sites are experiencing a similar issue, instead of being 
able to identify common issues across sites and obtain timely 
Solutions. In addition, maintainers at three sites and administrators at 
one site reported that recent ALIS software upgrades have resulted in 
the contractor not receiving ARs, with users unaware of this problem 
until they followed up on the ARs’ status. DOD officials told us they 
are aware of the issues surrounding the AR process and are 
collecting information on the types of ARs submitted from all sites. 
They stated that the largest types of ARs are related to data quality 


27 Naval Aviation Logistics Command Management Information System is an automated 
information system that provides aviation maintenance and material management 
personnel with information on which to base daily decisions by the Navy and the Marine 
Corps. 

oo 

An AR is a concern or question raised by a customer or user about any area of the F-35 
system, including ALIS. 

on 

The submitter of the AR prioritizes the request as either a Category I, which requires 
more immediate attention, ora Category II, and the contractor then prioritizes the requests 
to be addressed by technicians and engineers. Since the start of this process, 35,213 ARs 
have been submitted of which 34,113 have been closed—this can mean that the AR was 
either resolved or dismissed. Of the 897 ARs that were open as of January 2016, 42 were 
high-priority Category I ARs. 
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and integration management, and that this has been the case for the 
last 12 months. 

• ALIS has data accuracy and accessibility issues: ALIS users at all 
five sites we visited are concerned with data accuracy issues within 
the system, including missing or inaccurate data and inaccessibility of 
raw data within ALIS. Specifically, maintainers frequently have to 
resolve error messages for parts linked to electronic equipment logs 30 
that contain missing or inaccurate data when they try to fix a problem 
on the aircraft. Maintainers at two sites stated that recurring issues 
with electronic equipment logs have caused them to spend significant 
time resolving these issues instead of tending to other aircraft 
issues. 31 Additionally, they stated that parts requiring scheduled 
maintenance are not being tracked or updated accurately in ALIS. 
Program officials stated that these are life-limited parts that must be 
replaced by a certain time frame to avoid safety risks to the aircraft. 
To mitigate this issue, maintainers are currently logging this 
information outside of ALIS. Maintainers at Eglin Air Force Base said 
that they are spending 13 hours on average every day to track this 
information. Finally, maintainers at three sites stated they would like 
the ability to access raw data in ALIS to produce service-related 
reports. DOD officials stated that ALIS was designed to be used 
across services and, as such, reporting tools are not necessarily 
service-specific. However, ALIS users that operate the system daily 
continue to have issues with accessing the data required to keep 
aircraft mission-capable and generating service-specific reports for 
their squadrons. 

• Off-Board Mission Support and Training Management System 
applications are immature: Pilots and maintainers across all five 
sites we visited are concerned with the maturity and functionality of 
ALIS’s Off-Board Mission Support (OMS) and Training Management 
System (TMS) applications. OMS is a key application designed for 


30 Electronic equipment logs are linked to certain parts on the F-35. They contain critical 
information about the parts, such as date of manufacture and serial numbers. According to 
DOD officials, this information is supposed to be entered at the production facility and 
DOD is aware of these data inaccuracies. 

31 In an attempt to address this issue, DOD officials told us that the Defense Contract 
Management Agency recently informed the contractor about the recurring errors. 
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pilots to conduct essential mission planning and debriefing. 32 
Specifically, pilots at all five sites thought that the OMS application 
was poorly designed, cumbersome, and not user-friendly, especially 
with providing the necessary information they need to conduct their 
missions. Due to OMS’s current lack of functionality, pilots at two 
locations stated that they are forced to track vital mission planning 
information, information expected to reside within ALIS, outside of the 
system. According to the Office of the Director of Test and Evaluation, 
OMS’s lack of functionality could have an effect on combat missions 
and operational tempo. 33 

The Training Management System (TMS) is designed for pilots and 
maintainers to track training qualifications and assign personnel to 
carry out specific tasks based on their qualifications. However, pilots 
and maintainers at four sites told us that TMS is immature and does 
not function as intended. Maintainers at one site explained that TMS 
is supposed to keep track of maintainers’ and pilots’ qualifications 
and, based on that information, assign proper permission levels and 
Controls to a qualified maintainer to repair a problem on the aircraft. 
Instead, this is currently being tracked outside of ALIS, which is 
inefficiënt and could potentially result in this information not being 
populated back into the system. 

• Security risks exist: ALIS users cited concerns related to the 
system’s security. For example, pilots at one location explained that 
compact discs have to be recorded to move classified information 
from the aircraft into the classified network, rather than the system 
transmitting the information automatically—a practice that they said 
poses security risks. In addition, the ALOU and Central Point of Entry, 
as discussed earlier, are potential single points of failure and could be 
a security risk. A 2012 DOD Inspector General’s report 34 on ALIS also 
highlighted some security issues, including security accreditation and 
testing of hardware. Since that report, the F-35 program office has 
formed a team and developed a process to test, validate, and 
continuously monitor the security of ALIS applications and their 


op 

Mission planning supports simple training to complex combat scenarios and includes 
data such as navigation, threats, and weapons. 

33 Operational tempo is the rate of military actions or missions. 

34 Department of Defense Inspector General, Audit of F-35 Lightning II Autonomie 
Logistics Information System (ALIS). 
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interfaces with both military information networks and the contractor’s 
ALIS architecture. 


DOD Is Addressing ALIS 
Risks on a Case-by-Case 
Basis but Does Not Have 
a Plan to Prioritize and 
Address Them in a Holistic 
Manner 


DOD is aware of the risks identified by ALIS users, as well as others, and 
is addressing some on a case-by-case basis. However, DOD officials 
acknowledged that the department does not have a plan that would 
prioritize and address key risks in a holistic manner as program 
milestones approach. In recent years, the F-35 program has emphasized 
the criticality of ALIS to the success of the F-35. In October 2015, the 
F-35 Program Executive Officer stated that ALIS is a crucial component 
of the F-35 and should be treated as its own weapon system. Additionally, 
the Program Executive Officer stated that the program office changed its 
organizational structure to provide more senior leadership oversight of 
ALIS. Although more focus has been given to ALIS, according to DOD 
officials, the F-35 Program Executive Officer has reiterated that the focus 
is on completing the current development and testing of ALIS within the 
already established time frames, and with the previously planned funding. 
As a result, this has created an environment of competing priorities and 
limited resources for the entire program in the near term, including ALIS. 


ALIS users have identified key risks to ALIS’s functionality that we 
highlight in this report, and program office officials acknowledge that 
others may exist as welf The F-35 program office has taken some actions 
in an attempt to address smaller ALIS functionality issues between major 
software upgrades, and is considering the procurement of additional ALIS 
servers to add redundancy to the system. However, the current approach 
does not prioritize issues in a way that clearly designates which issues 
must be addressed within the time left in the system development and 
demonstration phase, and which issues could be addressed later as part 
of follow-on modernization. GAO guidance and DOD best practices 
emphasize that, prior to meeting key milestones, a plan to address 
specific risks that may be associated with major weapon acquisitions 
should be developed. Specifically, GAO’s Schedule Assessment Guide 
States that a high-quality and reliable acquisition Schedule includes the 
need to plan for and address major risks prior to meeting milestone 
dates. 35 DOD’s System Engineering Guide for System of Systems 
includes risk management as a key aspect of system engineering. It helps 
to ensure that program costs, schedules, and performance objectives are 


35 GAO-12-120G. 
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achieved at every stage in the life cycle and helps to communicate to all 
stakeholders the process for uncovering, determining the scope of, and 
managing program uncertainties. 36 Although key milestones—such as Air 
Force and Navy initial operational capability declarations and the start of 
full-rate production—are quickly approaching, DOD does not have a plan 
that prioritizes ALIS risks to ensure that the most important are 
expediently addressed and that ALIS is fully functional by these 
milestones. Furthermore, by continuing to address issues on a case-by- 
case basis, DOD risks that its solution to one issue could exacerbate 
another—for example, in addressing a security risk in isolation, DOD 
could inadvertently create further risks to data accessibility. According to 
F-35 program officials, a functional ALIS is key to the operational 
capability of the aircraft and the day-to-day ability to sustain the aircraft. 
Moreover, the department expects to significantly increase aircraft 
production within the next 5 years, so the number of aircraft that must be 
maintained and kept ready for flight will soon grow. By continuing to 
respond to issues on a case-by-case basis rather than in a holistic 
manner, there is no guarantee that DOD will address the highest risks, 
and as a result, DOD may encounter further Schedule and development 
delays, including system upgrades, which could affect operations and 
potentially lead to cost increases. 


DOD Has Estimated 
the Oosts of ALIS, but 
Including Additional 
Analyses and 
Information Would 
Increase the 
Credibility and 
Accuracy of lts 
Estimate 


DOD has estimated total ALIS costs to be approximately $16.7 billion 
over its 56-year life cycle. However, the estimate is not fully credible since 
DOD has not performed uncertainty and sensitivity analyses as part of its 
cost-estimating process. Moreover, while DOD has updated its estimate 
to be reflective of some program changes, it is also not fully accurate 
since DOD did not use historical cost data—both actual data from ALIS 
and data from comparable programs—when developing its ALIS 
estimate. Finally, other costs such as service customizations of ALIS may 
require additional future resources, and manual workarounds to the 
system currently require additional labor resources. 


36 Department of Defense, Systems Engineering Guide for System of Systems. 


Page 22 


GAO-16-439 F-35 Sustainment 



DOD Estimates that Total 
ALIS Costs Are about 
$16.7 Billion over a 56- 
Year Life Cycle 


DOD estimates that total ALIS costs are about $16.7 billion—about $562 
million to develop the system, about $1.1 billion to procure hardware and 
spare parts, and about $15.1 billion to sustain it in then-year dollars. 37 
DOD had expended approximately $505 million to develop ALIS as of 
December 2015, and the department estimates that continued 
development will oost an additional $57 million through 2017, which is 
when DOD expects ALIS 3.0 will be released for testing. In addition to the 
purchase oost for ALIS, DOD estimates that ALIS will oost about $15.1 
billion to sustain over a 56-year life cycle. Program officials told us that 
ALIS development will be completed within the planned resources, but 
that the system will require follow-on modernization and that the program 
office is currently planning for those additional costs. Table 2 provides 
more detail on ALIS oost elements. 


Table 2: Autonomie Logistics Information System (ALIS) Costs 

Cost element 

Then-year dollars in 
millions 

Development costs 

Expended as of December 2015 


505 

Estimated development costs remaining through 2017 


57 

Subtotal 


$562 

Estimated procurement costs 

Hardware 


931 

Spare parts 


147 

Subtotal 


$1,078 

Estimated sustainment costs 

Contractor support 3 


8,050 

Technology refresh 3 


3,850 

Hardware maintenance agreements 


1,603 

Software licensing agreements 


1,598 

Subtotal 


$15,101 

Total 


$16,741 


Source: GAO analysis of F-35 program office data. | GAO-16-439 


37 DOD’s estimate of ALIS sustainment costs includes operations and support costs—the 
costs of personnel, equipment, supplies, software, and services associated with operating, 
modifying, maintaining, supplying, and otherwise supporting the system. 
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Notes: Procurement and sustainment cost estimates are trom the F-35 program office 2014 cost 
estimate and span 56 years, from 2009 to 2065. Figures in table may not sum due to rounding. 
Development costs include both DOD’s and international partners’ share of developing ALIS. 
Procurement and sustainment costs represent only DOD’s share of estimated ALIS costs and do not 
include international partners’ estimated ALIS costs. 

a lncludes costs for ALIS administration and database maintenance as well as manpower for hardware 
replacement. 

b DOD’s definition of technology refresh is the periodic replacement of both custom-built and 
commercial-off-the-shelf System components, within a larger DOD weapon System, to ensure 
continued supportability throughout the weapon system’s life cycle. 

DOD’s Sustainment Cost 
Estimate for ALIS Is Not 
Fully Credible 

We found that the program office’s estimate for ALIS sustainment costs 
minimally met the best practices 38 for a credible cost estimate largely 
because it does not include uncertainty and sensitivity analyses (see app. 

II for more information on our assessment of the ALIS cost estimate). 
Every cost estimate contains a degree of uncertainty because of the 
many assumptions that must be made about the future. To mitigate this 
uncertainty, a variety of checks and analyses can be conducted to 
determine the credibility of the assumptions and the estimate as a whole. 
According to GAO’s Cost Estimating and Assessment Guide, cost 
estimates should include uncertainty analyses to determine the level of 
uncertainty associated with the estimate in order to be credible. A 
quantitative uncertainty analysis can provide a broad overall assessment 
of the risk in the cost estimate. In addition, credible cost estimates should 
include sensitivity analyses to examine how changes to individual 
assumptions and inputs affect the estimate as a whole. 

Although ALIS is a multibillion dollar system, it is not a formally 
designated stand-alone weapons system program; therefore, DOD is not 
required to perform a separate estimate of the system’s projected costs. 
ALIS is one cost element within the overall F-35 program, and the 
program office estimates that the system will constitute less than 2 
percent of the $891.1 billion total sustainment costs of the program. Cost 
estimators told us that despite its critical importance in operating and 
maintaining the entire F-35 fleet, since ALIS constitutes a small portion of 
the total estimate, it is not considered a major cost driver of the program. 
Therefore, cost estimators told us DOD’s cost-estimating guidance does 
not require them to perform uncertainty or sensitivity analyses for ALIS. 

As part of the planning process for F-35 sustainment, program officials 


38 These best practices are found in GAO’s Cost Estimating and Assessment Guide. See 
GAO-09-3SP 
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have examined the role of F-35 information systems (which includes 
ALIS) in relation to five major value streams—maintenance, supply chain, 
training, management, and sustaining engineering. While program 
officials said this process helped map the connections and 
interdependence between program elements, it did not quantify in 
uncertainty or sensitivity analyses the potential effects of ALIS on 
sustainment oosts specifically. 

Additionally, DOD did not perform analyses to determine how further ALIS 
Schedule delays or functionality issues could affect other F-35 costs. In 
lieu of the analyses, program officials stated that they assume that if ALIS 
does not perform as planned, aircraft could not be flown as frequently as 
intended, and this lower-than-expected utilization rate would therefore 
decrease sustainment costs. A 2013 DOD-commissioned study on 
reducing F-35 costs found areas of potential cost savings, but the 
implementation plan for this study also found that ALIS presents 
significant risk to the program. Particularly, this plan found that any 
functionality problems or Schedule slippage with ALIS will have a 
significant impact on costs—with downstream additional costs due to 
performance and Schedule delays potentially reaching up to $20-100 
billion. 39 The different conclusions drawn by the program office and this 
study suggest that DOD could better understand the effects that ALIS 
issues could have on overall program costs. Without performing 
uncertainty or sensitivity analyses, DOD will not understand how 
variabilities in ALIS-related assumptions could affect the estimate as a 
whole and the potential range of costs resulting from these variabilities. 


DOD’s ALIS Cost Estimate 
Is Not Fully Accurate, and 
Upcoming Program 
Changes May Affect 
Future Estimates 


We also assessed the ALIS estimate for accuracy and found that DOD 
partially met the standards for an accurate cost estimate. GAO’s Cost 
Estimating and Assessment Guide States a cost estimate should be 
based on historical data—both actual costs of the program and those of 
comparable programs—which can be used to challenge optimistic 
assumptions and bring more realism to a cost estimate. While we found 
that the DOD substantially met some best practices for an accurate cost 
estimate by properly adjusting for inflation and not including mathematical 
errors, the estimate uses contractor-provided data for material costs 


39 McKinsey & Company, F-35 O&S Cost Reduction Strategy (Sept. 17, 2013); and, JSF 
O&S Cost Reduction Implementation Plan (Sept 20, 2013). 
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instead of actual ALIS oosts or historical oost data from analogous 
programs that would make the estimate more accurate. Cost estimating 
officials said that they did not base their ALIS estimates on historical cost 
data because they believe that there are no programs analogous to ALIS. 
For example, there is a logistics system for the Air Force’s F-22 
program—also a fifth-generation aircraft—but officials stated that is far 
less complex than ALIS and does not include all of ALIS’s applications 
and intended functions. 40 However, multiple versions of ALIS have been 
fielded since 2010 and using historical data on known ALIS costs, as well 
as analogous data from the F-22 or other programs, would make the 
estimate more accurately representative of likely sustainment costs (see 
app. II for more Information on our assessment). 

GAO’s Cost Estimating and Assessment Guide also States that an 
estimate should be updated regularly to reflect significant changes in the 
program—such as when schedules or other assumptions change—so 
that it is always reflecting current status. The program office updates its 
estimate annually and incorporates program changes and evolving 
assumptions in these updates, documenting the changes from yearto 
year. However, the program office does not update all elements of the 
cost estimate. For example, technology refresh accounts for 
approximately 25 percent of ALIS sustainment costs and program officials 
were able to teil us the assumptions used to calculate these costs, 
however, they were unable to teil us, or identify within the estimating 
model, where these data came from or when their underlying 
assumptions were developed. 41 Additionally, over the course of our 
review, program officials highlighted some recent or upcoming program 
changes, such as the need for additional infrastructure, that were not 
included in their last estimate. 


40 GAO’s Cost Estimating and Assessment Guide States that cost estimators should collect 
actual cost data from a list of similar and legacy programs. Since most new programs are 
improvements of existing ones, data should be available that share common 
characteristics with the new program. The analogy cost estimating method uses actual 
costs from a similar program with adjustments to account for differences. A cost estimator 
typically uses this method when there is little actual cost data available but the technical 
and program definition is good enough to make the necessary adjustments and 
comparisons to a predecessor system. 

41 

DOD's definition of technology refresh is the periodic replacement of both custom-built 
and commercial-off-the-shelf system components, within a larger DOD weapon system, to 
ensure continued supportability throughout the weapon system’s life cycle. 
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The program office and services are exploring ways to decrease F-35 
sustainment oosts. For example, the services are taking different 
approaches to administering and maintaining ALIS, with the Marine Corps 
planning on using its own personnel, and the Air Force planning on using 
contractor support to administer the system, troubleshoot problems, and 
keep the system operating smoothly with software patches and other 
continuous improvements. Based on these service plans, the program 
office estimates that ALIS administration will oost more than $7 billion 
over the F-35 life cycle, in addition to about $1 billion for other contractor 
labor needs. Program officials said that these amounts may change 
based on upcoming sustainment decisions, including a potential way to 
decrease administrative oosts by establishing regional centers that could 
provide this support to a number of F-35 sites rather than having 
contracted administrators at each site. 

Other program changes have the potential to increase ALIS oosts and 
future estimates. Although DOD has included estimated oosts of ALIS 
technology refresh and software licensing, program officials have stated 
that the current estimate does not sufficiently capture the full oosts of 
follow-on modernization that will be required when upgrades or new 
versions of the system’s commercial off-the-shelf software are released. 
Because ALIS comprises multiple applications and interfaces with service 
networks and legacy information systems, engineers will have to 
reintegrate ALIS with other systems as well as the applications within 
ALIS in step with continuous upgrades and software improvements. 
Although DOD expects ALIS 3.0 to be the fully capable version that 
meets program requirements, officials at the program office and from all 
three services stressed that ALIS will not be a static system and that 
improvements and upgrades will not only be expected, but required, to 
keep ahead of technology obsolescence and evolve with emerging threat 
environments. Program officials stated that there is a need to build these 
follow-on modernization costs into the budgeting and cost-estimating 
processes. 

Program officials also stated that they may procure up to two more 
ALOUs for back-up and necessary redundancy and that, in addition to 
addressing the current risk of the single ALOU becoming a single point of 
failure, these additional ALOUs may facilitate greater government 
operation of ALIS and increase the potential for greater competition of 
future sustainment contracts. The program office bases much of its 
estimate on inputs from the services, such as their expected personnel 
needs and how they plan to operate and sustain their F-35 squadrons. 
The program plans to field one SOU for each F-35 squadron, and another 
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SOU at each forward operating location. Since deploying squadrons do 
not normally transit to the forward location together as one unit but rather 
in a more staggered fashion, having an SOU at both the squadron’s home 
base and another prepositioned at the forward deployed location would 
avoid potential problems of having a split squadron sustained with just 
one SOU. Having additional SOUs positioned at forward deployed 
locations would likely increase procurement costs and the downstream 
costs of maintaining and replacing them. 

While the program office has been incorporating some program changes 
and adjusting its cost estimating assumptions as the F-35 program grows 
and evolves, it is important to continue this effort in its annual estimate 
updates to reflect current or planned program changes such as those 
described above. Additionally, as the program gains more experience 
operating ALIS, using historical data—especially actual program costs as 
they become available—as the basis for its estimates can result in a more 
accurate and realistic picture of ALIS costs. Unless DOD’s estimate is 
based on an assessment of the most likely costs, the estimate may not be 
representative of how much it will cost to sustain ALIS and may inhibit 
informed decision making. 


Service Customizations 
and Workarounds Used to 
Overcome ALIS 
Limitations Could Require 
Additional Resources 


There are other ALIS-related costs that are not included in the estimate 
but that may place additional financial or manpower burdens on the 
services as ALIS continues to be concurrently developed and fielded. 
Services are responsible for bearing the costs of engineering any service- 
specific ALIS customizations. According to DOD officials, ALIS 
engineering changes must be agreed-upon by all services and partners in 
order to become new requirements. The program office then 
communicates these requested changes to the contractor software 
engineers developing ALIS. However, as ALIS users become more 
familiar with the system and its limitations, the services may request 
additional changes to the design of ALIS to meet their specific 
airworthiness and reporting requirements, and incur the additional costs 
to meet these needs. 


Some service-specific reporting requirements are being met through the 
use of time-consuming workarounds employed by ALIS users to 
compensate for current system limitations. According to program officials, 
some service-specific reporting requirements are not addressed by 
current ALIS functionality. For example, maintainers at several sites said 
that they must request maintenance data entered and housed within ALIS 
from the contractor because they cannot access this information 


Page 28 


GAO-16-439 F-35 Sustainment 




independently. They then use these raw data to generate reports on 
aircraft status using software programs outside of ALIS because the 
system does not have the capability to extract and process the data in the 
form that the services require. Program officials told us that they expect 
some of these workarounds to become unnecessary after upcoming 
system improvements. Other ALIS functionality issues that create the 
need for workarounds do not have scheduled Solutions, so both program 
and service headquarters officials are in the process of examining the use 
of these workarounds to determine whether they are truly necessary and 
will require an ALIS design change or whether service expectations 
should be better managed. Program officials told us that this workaround 
is not a problem with ALIS itself, but more an issue of the services 
requiring reports that the system was not designed to provide. Users at all 
the sites we visited told us that ALIS should have the functionality to 
create reports that they need. 

Program officials said that there is not an extra financial cost associated 
with these workarounds since they are performed by service personnel, 
but rather there is an opportunity cost to performing them—the additional 
time personnel spend on manually analyzing data and generating reports 
rather than performing their primaryjob duties. Neitherthe program office 
nor the services track the use of ALIS workarounds across all F-35 
squadrons, but squadron leadership at Eglin Air Force Base provided 
examples of workarounds performed at their site including the estimated 
amount of time personnel must spend to overcome ALIS shortcomings. 
They estimate that personnel at that location alone spend approximately 
150 man-hours per week on these workarounds—nearly the equivalent of 
having four full-time personnel dedicated to manually creating reports 
because, according to officials, ALIS currently cannot. Eglin Air Force 
Base personnel have created and updated 14 manual products mainly 
because of ALIS report limitations and because ALIS does not track 
information needed by senior leaders for unit-level analysis and 
subsequent maintenance and logistics decisions. They added that without 
these manual products, analysis, scheduling, and maintenance 
operations would suffer degradation and that mission accomplishment 
could be jeopardized. 

The use of workarounds varied across the F-35 sites we visited, and 
officials said that Eglin Air Force Base’s examples are not necessarily 
representative of other sites’ workarounds. They provide a sense of scale, 
however, for the labor burden placed on personnel because of current 
system immaturity and functionality issues. Personnel at F-35 sites we 
visited told us that the extra time they spend on ALIS workarounds 
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detracts from the time they have to maintain proficiency in their 
specialties, prevents them from coaching and training their subordinates, 
and decreases the amount of time they have to perform collateral duties. 
They added that ALIS is not yet fully autonomie and will require significant 
additional system improvements for it to perform to their expectations. 

The use of these ALIS workarounds highlights the current immature state 
of the system and underscores the need for DOD to prioritize, and 
address key ALIS risks as discussed earlier in this report. As the F-35 
program increases production over the next several years, sustained 
attention to addressing these issues and improving estimates of ALIS 
oosts can help decision makers better direct resources and ensure that 
ALIS meets the needs of its users. 


DOD Does Not Have 
a Program-Wide 
Training Plan for ALIS 
Users but Has Taken 
Some Initial Steps to 
Address Training 
Shortfalls 


DOD does not have a program-wide training plan for ALIS, but has taken 
initial steps to address training shortfalls. According to ALIS users at all 
five F-35 operational and training sites where we conducted our focus 
groups, training for ALIS is ineffective, and lacks a standardized, common 
curriculum for teaching users how to operate ALIS. Basic ALIS training 
courses are made available to all ALIS users at the F-35 Academie 
Training Center at Eglin Air Force Base. The training, according to ALIS 
users across all five sites, consists of a series of PowerPoint slides that 
are geared toward illustrating the conceptual nature of the system— 
showing how all the applications within ALIS are supposed to work— 
rather than how to actually operate the system as it currently exists. 
Furthermore, several ALIS users described the training as ineffective 
because it does not teach users how to explicitly operate specific ALIS 
applications. Forexample, the slideshow-based classroom training, which 
is optional for ALIS users, is not customized for the different users of 
ALIS. Pilots, maintainers, supply personnel, data analysts, and other F-35 
specialists are required to use certain ALIS applications, and often in 
different ways based on their job requirements. Maintainers at two of the 
five sites said that the classroom training reputation has become so bad 
that many new maintainers choose to skip the courses and proceed 
directly to on-the-job training when they begin working on the F-35. 


From the outset of the original ALIS release in 2010, DOD has relied on 
the contractor to manage all ALIS training. ALIS training is currently 
heavily dependent on learning on the job, which consists of users learning 
how to operate ALIS from colleagues and through trial and error. 
According to DOD officials, this is not an uncommon practice for new 
systems within the department; however, those officials agree that 
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training typically begins with a common classroom curriculum, and is 
supplemented with on-the-job-training. Almost every ALIS user in our 
focus groups noted that they do not learn how to operate any ALIS 
applications until on-the-job-training begins on theflight line. Specifically, 
the classroom training does not afford an opportunity to practice on 
“sandbox” or “ghost” systems that would simulate how ALIS is used on a 
daily basis by users. Instead, the practice comes in a live operational 
environment, where basic ALIS functions and practices are taught by 
supervisors or other users within the respective squadron. According to 
ALIS users at all five sites, this practice has led to users learning 
inconsistent methods and shortcuts for maneuvering through the different 
ALIS applications. In most cases, these practices differ between 
squadrons and services; however, there were also cases that some 
maintainers highlighted where users were learning different practices 
within the same squadron. ALIS users reported that they have created 
different workarounds to overcome ALIS functionality issues, but have 
also cited instances of not using ALIS as intended because the system is 
unwieldy and time-consuming. ALIS users said they are learning to 
operate the system in different ways and then perpetuating these 
methods, creating a situation where ALIS may not be operated in the 
most effective, efficiënt, or up-to-date way across all F-35 sites. 
Furthermore, they are learning to use the system in a live operational 
environment, running the risk of making errors that could ultimately affect 
aircraft availability. 

Program officials acknowledged the training shortfalls identified in the 
focus-group sessions. In response to these shortfalls, the program office 
has taken some initial steps to address some of the issues. Specifically, 
the program office, in concert with the contractor, has developed Mobile 
Training Teams to offer a way to train ALIS users outside of the F-35 
Academie Training Center and at their specific F-35 sites. These teams 
are deployed to F-35 operational and testing sites to help keep ALIS 
users up-to-date with ALIS software version releases, which, according to 
ALIS users at all sites we visited, had been a significant problem with 
ALIS training. Specifically, according to ALIS users at all these sites, ALIS 
trainers and administrators are rarely up-to-date on the latest ALIS 
releases and functionality changes; therefore, it has led to inconsistencies 
in teachings and practices at the squadron level. Mobile Training Teams 
offer ALIS version-specific training at each site based on a sequencing 
Schedule developed by the program office. 

In addition, the program office has rolled out an ALIS Training Evaluation 
to determine the current state of ALIS training across all F-35 sites. The 
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end goal of this evaluative process is to identify underlying training 
deficiencies, and to develop corrective courses of action to mitigate these 
deficiencies. The process will include a series of site visits to include 
course audits, interviews, curricula inspections, and stakeholder surveys 
as applicable for root-cause analysis. As of January 2016, a team within 
the program office had just begun this process; therefore, they could not 
provide any details beyond the effort’s scope, methodology, and 
associated time frames. 

According to best practices of information-technology training, effective 
training of users is essential to the workforce supporting an information- 
technology system. 42 The practices suggest that entities develop 
Strategie Learning Plans, or Overarching Training Plans, to help align 
training programs with priorities. Furthermore, as part of this process, the 
practices state that it is important that the training design and delivery 
process ensures learning occurs during the training and also ensures that 
the user applies the training on the job. Additional guidance on strategie 
training in the federal government States that training plans can aid in the 
performance of government programs. 43 Specifically, a training plan can 
present a business case for proposed training and development 
investments, including the identified problem or opportunity, the concept 
for an improved situation or condition, and linkages with strategie 
objectives. 

According to DOD officials, ALIS training has been a difficult process to 
manage because of the dynamic nature in which ALIS has been 
developed and upgraded since its initial release. Because new versions 
of ALIS have been regularly released in a staggered manner across 
F-35 sites, they said it has been difficult to sufficiently train all ALIS users 
on the most up-to-date versions and teach consistent practices. However, 
with only one major version upgrade remaining prior to version 3.0 (the 
final major software release), issues related to constantly changing the 
system should decrease. A standardized, program-wide training plan 
could remove the emphasis from on-the-job training and provide a 
comprehensive, standardized training curriculum across the program. 
Without a program-wide training plan that assures that consistent learning 
occurs in the classroom, and is then applied by users on the job, the 


42 GAO-03-390. 

43 GAO-04-546G. 
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program runs the risk of continuing to allow users to learn irregular or 
incorrect practices through a training culture driven by on-the-job training, 
which could impact aircraft availability and safety. 


Conclusions 


The F-35 program is DOD’s largest and most costly acquisition program 
to date. According to senior defense officials, not only does the F-35 
program represent the future of tactical fighter aircraft in our military, but it 
is also vital to the security of our nation moving forward. With the Marine 
Corps already having declared the F-35 both deployable and combat- 
ready, the Air Force and Navy set to do the same within the next 2 years, 
and the full-rate production of the aircraft set for 2019, including the ramp 
up of sustainment activities, it is imperative that DOD address major risks 
associated with its central logistics system—ALIS. The program office has 
taken steps to identify risks associated with ALIS, including all of those 
identified by participants in our focus groups—and has begun, on a case- 
by-case basis, to address some of these risks; however, without a plan to 
prioritize risks and address them in a systematic and holistic manner, 

DOD runs the risk of having an ALIS that is not fully functional as it 
approaches key program milestones. Without a fully functional ALIS, 

DOD could face operational and Schedule risks and potential cost 
increases to a program that is already the most expensive in DOD’s 
history. 

Although DOD has estimated the costs of ALIS, additional information 
would increase the accuracy and credibility of the estimate. While ALIS is 
projected to constitute less than 2 percent of the $891.1 billion total 
sustainment costs of the program, the financial impact that nonfunctional 
aspects of ALIS may have on the overall operations and sustainment of 
the aircraft could be significant. Until DOD does more analyses to 
determine the impact ALIS has (and the impact a nonfunctional ALIS 
could have) on overall sustainment costs, DOD will not know how much 
the costs of ALIS and overall sustainment could fluctuate. Furthermore, 
while the program office has been incorporating some program changes 
and adjusting its cost-estimating assumptions for ALIS, it is important for 
the program office to improve the reliability of its estimate by using 
historical data. It is also important for DOD to incorporate program 
changes that will likely affect ALIS costs in future estimates, such as 
decisions to enhance ALIS infrastructure or decrease planned numbers of 
administrative personnel with regional support centers. 

Finally, although DOD has recognized that ALIS training needs 
improvement and has made some temporary fixes to address the current 
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shortcomings, DOD has yet to develop a program-wide training plan that 
would take the focus off an almost explicit on-the-job-training approach, 
and provide greater consistency among ALIS users. Considering the 
importance ALIS has to the operations and sustainment of the aircraft, 
and that DOD plans to purchase and operate nearly 500 more F-35s in 
the next 5 years, it will be important that ALIS users be on the same page 
with regard to operating the system. Further training inconsistencies could 
lead to impacts on aircraft availability and safety. 


Recommendations for 
Executive Action 


We recommend that the Secretary of Defense direct the F-35 Program 

Executive Officer to take the following four actions: 

• To ensure that risks associated with ALIS are addressed expediently 
and holistically, develop a plan that would prioritize and address ALIS 
issues, prior to the start of full-rate production for the program. 

• To improve the reliability of its cost estimates, conduct uncertainty and 
sensitivity analyses consistent with cost-estimating best practices 
identified in GAO’s Cost Estimating and Assessment Guide. 

• To improve the reliability of its cost estimates, ensure that future 
estimates of ALIS costs use historical data as available and reflect 
significant program changes consistent with cost-estimating best 
practices identified in GAO’s Cost Estimating and Assessment Guide. 

• To ensure that ALIS training issues are fully addressed, develop a 
standardized, program-wide plan for ALIS training through the life 
cycle of the program. 


Agency Comments 
and Our Evaluation 


In written comments on a draft of this report, DOD concurred with two 
recommendations and partially concurred with two recommendations. 
DOD’s comments are summarized below and reprinted in appendix III. 
DOD also provided technical comments, which we have incorporated into 
our report where appropriate. 

DOD concurred with the recommendation that the Secretary of Defense 
direct the F-35 Program Executive Officer to develop a plan that would 
prioritize and address ALIS issues, prior to the start of full-rate production 
for the program. DOD stated that the F-35 program office began 
developing an ALIS Technical Roadmap in early 2016. The department 
added that at its completion later in 2016, this roadmap will be the 
foundation of a plan to identify, document, and prioritize ALIS risks; 
address them holistically; and inform budget priorities, as the program 
transitions from development into sustainment and follow-on 
modernization. Additionally, to mitigate the risk of single-point failures in 
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the infrastructure, the program office contracted to acquire backup ALIS 
hardware in 2015; the backup hardware will be operational by early 2017. 
We state in our report that the department was aware of the risks to ALIS 
and believe that if DOD develops a plan to identify, document, and 
prioritize ALIS risks, address them holistically, and inform budget priorities 
prior to full-rate production, this action should address our 
recommendation. We further state in our report that DOD was in the early 
stages of acquiring backup ALIS hardware to mitigate the risk of single- 
point failures in the infrastructure, but had not yet allotted funding. We 
believe this backup system will be critical as the program approaches full- 
rate production. 

DOD partially concurred with the recommendation that the Secretary of 
Defense direct the F-35 Program Executive Officer to conduct uncertainty 
and sensitivity analyses consistent with cost-estimating best practices 
identified in GAO’s Cost Estimating and Assessment Guide. DOD stated 
that the department considers the sensitivity analyses that the F-35 
program office performs to be a form of uncertainty analysis, as described 
in DOD’s Cost Assessment and Program Evaluation Operating & Support 
Cost Estimating Guide ; however, the DOD cost estimating guidance does 
not require DOD to conduct a sensitivity or uncertainty analysis on ALIS 
since DOD does not consider ALIS a major cost driver of the F-35 
program. As our report States, according to GAO’s Cost Estimating and 
Assessment Guide, cost estimates should include uncertainty analyses to 
determine the level of uncertainty associated with the estimate in order to 
be credible. A quantitative uncertainty analysis can provide a broad 
overall assessment of the risk in the cost estimate. A sensitivity analysis 
can examine how changes to individual assumptions and inputs affect the 
estimate as a whole. Although the F-35 program office may conduct 
analyses consistent with DOD cost estimating guidance, it has not 
conducted an uncertainty or sensitivity analysis specifically for ALIS. 
Although ALIS is projected to constitute less than 2 percent of the $891.1 
billion total sustainment costs of the program, the financial impact that 
nonfunctional aspects of ALIS may have on the overall operations and 
sustainment of the aircraft could be significant. For example, a 2013 
DOD-commissioned plan found that any functionality problems or 
Schedule slippage with ALIS will have a significant impact on costs—with 
downstream additional costs due to performance and Schedule delays 
potentially reaching up to $20-100 billion. We continue to believe that 
without completing uncertainty and sensitivity analyses to determine the 
effect ALIS has (and the impact a nonfunctional ALIS could have) on 
overall sustainment costs, DOD will not know how much the costs of ALIS 
and overall sustainment could fluctuate. 


Page 35 


GAO-16-439 F-35 Sustainment 



DOD partially concurred with the recommendation that the Secretary of 
Defense direct the F-35 Program Executive Officer to ensure that future 
estimates of ALIS oosts use historical data as available and reflect 
significant program changes consistent with cost-estimating best 
practices identified in GAO’s Cost Estimating and Assessment Guide. 
DOD stated that the department will ensure that the future F-35 program 
ALIS cost estimates continue to use the latest available historical cost 
data as appropriate and reflect the latest approved technical baseline 
when the program office incorporates these into the program of record, 
according to DOD’s Cost Assessment and Program Evaluation Operating 
and Support Cost Estimating Guide; however, we found that DOD was 
not using all available historical data. As our report States, according to 
GAO’s Cost Estimating and Assessment Guide, a cost estimate should 
be based on historical data—both actual costs of the program and those 
of comparable programs—which can be used to challenge optimistic 
assumptions and bring more realism to a cost estimate. While we found 
that DOD substantially met some best practices for an accurate cost 
estimate by properly adjusting for inflation and not including mathematical 
errors, the estimate uses contractor-provided data for material costs 
instead of actual ALIS costs or historical cost data from analogous 
programs that would make the estimate more accurate. We continue to 
believe that it is important for the program to improve the reliability of its 
ALIS estimate by using historical data to the greatest extent possible. It is 
also important for DOD to incorporate program changes that will likely 
affect ALIS costs in future estimates, such as decisions to enhance ALIS 
infrastructure or decrease planned numbers of administrative personnel 
for regional support centers. 

DOD concurred with the recommendation that the Secretary of Defense 
direct the F-35 Program Executive Officer to develop a standardized, 
program-wide plan for ALIS training through the life cycle of the program. 
DOD stated that, to address immediate issues, the F-35 program office 
deployed Mobile Training Teams to assist ALIS users at their home base 
locations, and to address longer-term issues, the program office began a 
comprehensive evaluation of ALIS training in 2015. According to DOD, 
the completion of this evaluation in 2016 will inform development of a plan 
to address long-term ALIS training issues. We agree and state in our 
report that the F-35 program has taken some positive steps to address 
short-term training shortfalls by deploying Mobile Training Teams as a 
way to train ALIS users outside of the F-35 Academie Training Center. 

We also report that the program has recently begun a comprehensive 
evaluation of ALIS training to determine the current state of ALIS training 
across all F-35 sites. If the F-35 program leverages this comprehensive 
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evaluation, when it is completed, to develop a program-wide plan for ALIS 
training through the life cycle of the program, this action should address 
our recommendation. 


We are sending copies of this report to appropriate congressional 
committees; the Secretary of Defense; the Secretaries of the Air Force, 
Army, and Navy; and the Commandant of the Marine Corps. In addition, 
the report is available at no charge on the GAO website at 

http://www.gao.gov. 

If you or your staff has any questions about this report, please contact me 
at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last page 
of this report. Staff members making key contributions to this report are 
listed in appendix IV. 




Cary Russell, Director 

Defense Capabilities and Management 
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To determine the extent to which the Department of Defense (DOD) has a 
plan to ensure the Autonomie Logistics Information System (ALIS) is fully 
functional as the key F-35 program milestones approach, we reviewed 
documentation of program plans with relevant sustainment elements 
including the F-35 Global Sustainment Plan, the Weapon System 
Planning Document, the F-35 Autonomie Logistics Global Sustainment 
Concept of Operations, and the F-35 Operational Requirements 
Document. We also selected and conducted site visits at a 
nongeneralizable sample of five F-35 operational and training sites: 

• Eglin Air Force Base, Florida 

• Luke Air Force Base, Arizona 

• Edwards Air Force Base, California 

• Nellis Air Force Base, Nevada 

• Marine Corps Air Base Yuma, Arizona 

We selected these sites in consultation with service officials to ensure we 
obtained perspectives across all three services and at both operational 
and testing sites. 1 During these visits, we convened 17 non-generalizable 
focus-group sessions with a range of ALIS users from all three services to 
obtain information on the operability and deployability of ALIS, and how 
any ALIS issues may pose risks for F-35 operations and sustainment. 
Specifically, we convened groups of maintainers, pilots, system 
administrators, and trainers. We also held focus groups with contractor 
personnel responsible for training and administering ALIS at these sites. 
There were approximately a total of 120 participants in these focus 
groups. Table 3 includes a breakdown of the focus groups we held at the 
various locations. 


^hese locations included Air Force and Marine Corps bases. In addition, although we did 
not visit a Navy installation, we spoke with Navy personnel working in the F-35 program at 
one of these locations. 
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Table 3: Focus Groups Convened at F-35 Sites 


Location 

Focus groups convened 

Eglin Air Force Base 

• Pilots, maintainers, trainers, administrators 

Luke Air Force Base 

• Pilots, maintainers, trainers, administrators 

Marine Corps Air Base Yuma 

• Pilots, maintainers, administrators 

Edwards Air Force Base 

• Pilots, maintainers, administrators 

Nellis Air Force Base 

• Pilots, maintainers, administrators 


Source: GAO I GAO-16-439 


We worked with our methodologist to develop a focus group script that 
included questions across four main categories: Training, ALIS Positives, 
ALIS Negatives, and Risks, that was used at all five site visits. For 
consistency, our methodologist facilitated all focus group sessions. To 
analyze the focus-group responses, we conducted content analyses, 
developing categories and sub-categories and coding comments from 
each focus group to these categories. After each comment had been 
coded by an analyst, another analyst independently reviewed each code 
and either agreed or disagreed with the coding decision. Where there was 
disagreement in the coding decision, the two analysts discussed it and 
came to a resolution. Based on the content analysis, we described the 
overarching ALIS benefits and risks obtained from ALIS users about the 
system and any risks it poses to the F-35 program as it approaches key 
milestones. After obtaining information on DOD’s current approach to 
addressing the functionality issues that ALIS users identified, we 
evaluated information we obtained for consistency with best practices 
from GAO’s Schedule Assessment Guide and DOD’s System 
Engineering Guide for System of Systems that provide guidance and best 
practices on how, prior to meeting key milestones, a plan to address 
specific risks that may be associated with major weapon acquisitions 
should be developed. 2 We also interviewed key DOD and contractor 
officials to collect information about building and testing ALIS, the 
capabilities of ALIS, metrics collected on ALIS’s development and 
performance, software upgrades to ALIS, and how the 
F-35 program office is addressing ALIS functionality issues. 


2 GAO, Schedule Assessment Guide, GAO-12-120G (Washington, D.C.: May 2012) and 
Department of Defense, Systems Engineering Guide for System of Systems (Washington, 
D.C.: August 2008). 
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To determine the extent to which DOD has credibly and accurately 
estimated ALIS costs, we evaluated the reliability of DOD’s estimate of 
ALIS costs contained in the F-35 program office’s 2014 estimate of F-35 
operating and support (O&S) costs, the most up-to-date estimate 
completed at the time of our review. The program office completed the 
2014 cost estimate in the spring of 2015 and plans to release its 2015 
updated estimate in spring of 2016. The Office of the Directorfor Cost 
Assessment and Program Evaluation (CAPE) also performed an F-35 
O&S cost estimate in 2013, but has not updated it. Both CAPE and Joint 
Program Office officials told us they used the same cost inputs, 
methodology, and ground rules and assumptions when estimating ALIS 
costs in their respective estimates. We used characteristics contained in 
GAO’s Cost Estimating and Assessment Guide to assess the reliability of 
DOD’s estimate of ALIS costs. 3 According to the guide, there are four 
general characteristics of sound cost estimating: being well-documented, 
comprehensive, credible, and accurate. Forthe purposes of this review, 
we conducted a limited assessment and used two of the four general 
characteristics of sound cost estimating included in this guide: being 
credible and accurate. We chose these characteristics because ALIS 
costs represent only one element of the total F-35 cost estimate—less 
than 2 percent of projected F-35 sustainment costs—and therefore we 
determined that it would not be appropriate to assess whether the 
estimate was comprehensive or well-documented. To determine whether 
the credible and accurate characteristics were met, we reviewed 
documentation used to generate the program office’s estimate, including 
data sources, assumptions, and cost models, and we interviewed cost- 
estimating officials from the program office and CAPE. We also 
interviewed other officials from the program office and service 
headquarters to discuss the cost effects of ALIS Schedule delays and 
development issues. Results of our assessment of the estimate’s 
credibility and accuracy, along with descriptions of these characteristics 
and their associated best practices, are detailed in appendix II of this 
report. We found the data to not be fully reliable, which we discuss in 
further detail in the report and in appendix II. 


3 GAO, Cost Estimating and Assessment Guide: Best Practices for Developing and 
Managing Capital Program Costs , GAO-09-3SP (Washington, D.C.: March 2009). The 
methodology outlined in the Cost Estimating and Assessment Guide is a compilation of 
best practices thatfederal cost-estimating organizations and industry use to develop and 
maintain reliable cost estimates throughout the life of an acquisition program. 
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Finally, to determine the extent to which DOD has developed a plan to 
manage ALIS training for users, we reviewed key documentation related 
to ALIS and F-35 training, and used information trom our focus-group 
sessions across various types of ALIS users from all three services to 
obtain information on the current state of ALIS training. We also 
interviewed key DOD and contractor officials. We evaluated all of the 
information we received using GAO-developed and industry best 
practices for information-technology training, and DOD’s Policies and 
Procedures for Acquisition of Information Technology. 4 

To address all of our objectives, we collected and analyzed information 
and interviewed officials from the following Department of Defense (DOD) 
offices: 

• Office of the Under Secretary of Defense (Acquisitions, Technology 
and Logistics) 

• Office of the Director for Oost Assessment and Program Evaluation 
(CAPE) 

• Office of the Director for Operational Test and Evaluation (DOT&E) 

• Department of the Air Force 

• Department of the Navy 

• Headquarters Marine Corps 

• F-35 Joint Program Office 

We also collected and analyzed information and interviewed officials from 
Lockheed Martin in Fort Worth, Texas, and Orlando, Florida. 

We conducted this performance audit from April 2015 to April 2016 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficiënt, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides and reasonable basis for our findings and 
conclusions based on our audit objectives. 


4 GAO, Information Technology Training: Practices of Leading Private-Sector Companies, 
GAO-03-390 (Washington, D.C.: Jan. 31,2003); and Human Capital: A Guide for 
Assessing Strategie Training and Deveiopment Efforts in the Federal Government, 
GAO-04-546G (Washington, D.C.: March 2004). 
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Program Office’s 2014 Autonomie Logistics 
Information System (ALIS) Operating and 
Support (O&S) Cost Estimate 


We completed an assessment of the ALIS oosts in the F-35 program 
office overall F-35 O&S cost estimate on the basis of two 
characteristics—being credible and accurate—and their associated best 
practices derived from the GAO Cost Estimating and Assessment Guide. 
After reviewing documentation that the program office submitted tor its 
2014 F-35 ALIS O&S cost estimate, conducting interviews with program 
office cost-estimating officials, and reviewing relevant sources, we 
determined that these cost estimates are not fully reliable. While we found 
that the program office estimate tor ALIS is partially accurate, the 
estimate is minimally credible. These evaluations are shown in table 4. 

We determined the overall assessment rating by assigning each 
individual best practice rating a number: Not Met = 1, Minimally Met = 2, 
Partially Met = 3, Substantially Met = 4, and Met = 5. Then, we took the 
average of the individual best practice assessment ratings to determine 
the overall rating for each of the two characteristics. The resulting 
average becomes the Overall Assessment as follows: Not Met = 1.0 to 
1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4, Substantially 
Met = 3.5 to 4.4, and Met = 4.5 to 5.0. A cost estimate is considered 
reliable if the overall assessment ratings for each of the two 
characteristics are substantially or fully met. If any of the characteristics 
are not met, minimally met, or partially met, then the cost estimate does 
not fully reflect the characteristics of a high-quality estimate and cannot 
be considered reliable. 


Table 4: Summary Assessment of F-35 Program Office Estimate of Autonomie Logistics Information System (ALIS) Costs 


Characteristic 

Overall 

assessment 

Best practice 3 

Individual 

assessment 

Credible 

Minimally met 

The cost estimate includes a sensitivity analysis that identifies a range of 
possible costs based on varying major assumptions, parameters, and 
data inputs. 

Not met 



A risk and uncertainty analysis was conducted that quantified the 
imperfectly understood risks and identified the effects of changing key 
cost driver assumptions and factors. 

Not met 



Major cost elements were cross-checked to see whether results were 
similar. 

Not met b 



An independent cost estimate was conducted by a group outside the 
acquiring organization to determine whether other estimating methods 
produce similar results. 

Partially met c 

Accurate 

Partially met 

The cost estimate results are unbiased, not overly conservative or 
optimistic and based on an assessment of most likely costs. 

Not met 



The estimate has been adjusted properly for inflation. 

Substantially met 



The estimate contains few, if any, minor mistakes 

Substantially met 
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Office's 2014 Autonomie Logistics Information 

System (ALIS) Operating and Support (O&S) 

Cost Estimate 

Overall 

Characteristic assessment 

Best practice® 

Individual 

assessment 


The cost estimate is regularly updated to reflect significant changes in 
the program so that it is always reflecting current status. 

Substantially met 


Variances between planned and actual costs are documented, 
explained, and reviewed. 

n/a d 


The estimate is based on a historical record of cost estimating and actual 
experiences from other comparable programs. 

Minimally met 


The estimating technique for each cost element was used appropriately. 

Minimally met® 


Source: GAO analysis of F-35 program office data. | GAO-16-439 


Note: n/a = not applicable. 

a Best practices are derived from GAO, Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). 

b The F-35 program office told us it uses cross-checks, but we could not confirm this through analysis 
of the documentation it provided. 

c The Department of Defense’s (DOD) Office of the Directer for Cost Assessment and Program 
Evaluation (CAPE) conducted an independent cost estimate of the F-35 program in 2013 that 
includes ALIS costs, but this estimate has not been updated since then. 

d The F-35 program office did not provide actual cost data as a basis for the estimate and we were, 
therefore, unable to perform variance analysis. 

e The F-35 program office did not provide data that would allow us to determine the appropriateness of 
the methodology used to estimate ALIS costs. 
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Appendix III: Comments from the 
Department of Defense 


ASSISTANT SECRETARY OF DEFENSE 

3500 DEFENSE PENTAGON 
WASHINGTON, OC 20301 -3500 


MAÏ! 2 5 2016 


Mr. Cary Russell 

Director, Defense Capabilities and Management 
U.S. Government Accountability Office 
441 G Street, N.W. 

Washington, DC 20548 

Dear Mr, Russell: 

This is the Department of Defense (DoD) response to the Government Accountability 
Office (GAO) Draft Report, GAO-16-439, “F-35 SUSTAINMENT: DOD Needs a Plan to 
Address Risks Related to lts Central Logistics System” dated March 14, 2016 (GAO Code 
352004). Detailed comments on the report recommendations are enclosed. 


Sincerely, 



David J. Berteau 


Enclosure: 
As stated 



LOGISTICS AND 
MATERIEL READINESS 
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GAO Draft Report Dated March 14, 2016 
GAO-16-439 (GAO CODE 352004) 

“F-35 SUSTAINMENT: DOD NEEDS A PLAN TO ADDRESS RISKS RELATED TO ITS 
CENTRAL LOGISTICS SYSTEM” 

DEPARTMENT OF DEFENSE COMMENTS 
TO THE GAO RECOMMENDATIONS 


RECOMMENDATION 1: To ensure that risks associated with ALIS are addressed expediently 
and holistically, the GAO recommends that the Secretary of Defense direct the F-35 Program 
Executive Offreer develop a plan that would prioritize and address ALIS issues, prior to the start 
of full-rate production for tlie program. 

DoD RESPONSE: Concur 

Tlie Department concurs with the recommendation to address ALIS risks. The F-35 Program 
Executive Officer (PEO) began developing an ALIS Technical Roadmap in early 2016. At its 
completion later in 2016, this effort will be the foundation of a plan to identify, document and 
prioritize ALIS risks, address them holistically and inform budget priorities, as the program 
transitions from development, into sustainment and follow-on modemization. Additionally, to 
mitigate the risk of single point failures in the infrastructure, the PEO contracted to acquire 
baekup ALIS hardware in 2015; the backup hardware will be operational by early 2017. 

RECOMMENDATION 2: To improve the reliability of its cost estimates, the GAO 
recommends that the Secretary of Defense direct the F-35 Program Executive Officer conduct 
uncertainty and sensitivity analyses consistent with cost estimating best practices identified in the 
GAO Cost Estimating and Assessment Guide. 

DoD RESPONSE: Partially Concur 

The Department partially concurs with the recommendation to improve the reliability of its cost 
estimates through uncertainty and sensitivity analysis. The Department considers the sensitivity 
analyses that the F-35 PEO regularly perfonns to be a fonn of uncertainty analysis, as described 
in the 2014 Office of the Secretary of Defense (OSD), Cost Assessment and Program Evaluation 
(CAPE) Operating & Support (O&S) Cost-Estimating Guide, Section 5.3.5. The Department 
will ensure that PEO continues to perform sensitivity analyses accordingto DoD guidance. 

RECOMMENDATION 3: To improve the reliability of its cost estimates, the GAO 
recommends that the Secretary of Defense direct the F-35 Program Executive Officer ensure that 
future estimates of ALIS costs use historical data as available and reflect significant program 
changes consistent with cost estimating best practices identified in the GAO Cost Estimating and 

Assessment Guide. 

DoD RESPONSE: Partially Concur 

The Department partially concurs with the recommendation to improve the reliability of its cost 
estimates through using historical data and incorporating program changes. The Department will 


0 
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ensure that future F-35 PEO ALIS cost estimates continue to use the latest available historical 
cost data as appropriate and reflect the latest approved technical baseline when the PEO 
incorporates these into the program of record, according to DoD guidance in the 2014 OSD 
CAPE O&S Cost-Estimating Guide. 

RECOMMENDATION 4: To ensure that ALIS training issues are fully addressed, the GAO 
recommends that the Secretary of Defense direct the F-35 Program Executive Officer develop a 
standardized, program-wide plan for ALIS training through the life cycle of the program. 

DoD RESPONSE: Concur 

The Department concurs with the recommendation to address ALIS training issues. To address 
immediate issues, the F-35 PEO deployed Mobile Training Teams to assist ALIS users at their 
home base locations. To address longer-tenn issues, the PEO began a comprehensive evaluation 
of ALIS training in 2015. At its completion in 2016, the evaluation will infonn development of 
a plan to address long-tenn ALIS training issues. 


1 
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